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ABSTRACT 

This paper focuses on the design of a serious game for the 
teaching and assessment of generic competences, placing 
particular emphasis on the competences assessment aspect. 
Taking into account important aspects of competence assessment 
such as context, feedback and transparency, among other 
aspects, and using the University of Deusto's Generic 
Competences Assessment Model based on the defining of levels, 
indicators and descriptors as a reference point, a serious game 
has been designed for the development and evaluation of two 
Generic Competences: Problem Solving and Entrepreneurship, 
aimed at final-year undergraduate and first-year postgraduate 
students. The design process shows that having a Competence 
Assessment Model based on levels, indicators and descriptors is 
of great help in defining the game's scenarios and learning and 
assessment activities. Serious games can also be excellent 
resources to help in the development and assessment of generic 
competences, but not as a unique tool, since the concept of 
competence in itself is highly complex (integrating knowledge, 
skills, attitudes and values) and some elements might require 
other methods and techniques for its development. It also reveals 
the difficulties of evaluating competences in general and through 
serious games in particular. 

KEYWORDS: EVALUATION, ASSESSMENT, SKILLS, 

COMPETENCES, VIDEO GAMES 


1 INTRODUCTION 

This paper discusses the design of a serious game for the 
development of generic competences in terms of evaluation, i.e., 
it focuses on how to evaluate generic competences via a serious 
game. Starting from a theoretical analysis of competence 
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assessment in general, it moves on to the design of the 
evaluation strategy of two Generic Competences (Problem 
Solving and Entrepreneurship) in a serious game being 
developed by the University of Deusto, and is aimed at final- 
year undergraduate and first-year postgraduate students. The 
University of Deusto's Competence Assessment Model, based on 
a structure of levels, indicators and descriptors, has been 
followed in order to design the evaluation strategy in the serious 
game. The study is part of a project which is in process of 
development, being at the present moment at the pilot 
experimentation phase. 

2 COMPETENCE-BASED LEARNING AND 
ASSESSMENT 

2.1 Competence-Based Learning 

Competence-based learning is an approach to learning and 
teaching that focuses on the development of generic or 
transversal competences (common to different study programs) 
and specific competences (belonging to each individual study 
program) that allow a person to gain scientific knowledge and 
techniques for their career, apply them in diverse and complex 
contexts, and incorporate them into their own attitudes and 
values in their own personal and professional way of acting 
(Villa & Poblete, 2008). This approach has progressively been 
introduced into European universities since the Bologna 
Declaration (1999) and is giving rise to significant 
methodological and organizational changes within universities. 
Competence-based learning entails passing from the teaching 
paradigm to the learning paradigm, from an education system 
based on the transmission of knowledge to an education system 
that places students' learning at the educational center of its 
action (Legault, 2012; Villa, Gonzalez, Auzmendi, Bezanilla & 
Laka, 2003). 

There are several definitions of Competence. Villa & Poblete 
(2008) see competence as “good performance in diverse, 
authentic contexts based on the integration and activation of 
knowledge, rules and standards, techniques, procedures, abilities 
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and skills attitudes and values” (p. 29). The Tuning Project 
(2008, p. 139) concludes that “competences represent a dynamic 
combination of cognitive and metacognitive skills, knowledge 
and understanding, interpersonal, intellectual and practical skills, 
and ethical values”. After reviewing various definitions of the 
term “competence”, Rodriguez & Vieira (2009) concluded that 
competences: 

—Are based on action in order to respond successfully to a 
demand or purpose 

—Are linked to a context 

—Are verifiable, and they can be learned and evaluated 

—Mobilize and integrate different elements: knowledge, 
skills, procedures and attitudes. 

There is often no distinction made between competence and 
skill in existing literature, and sometimes different terms are 
used to refer to the same concept. “Transferable skills”, “general 
skills”, “key skills”, “common skills” are defined differently by 
different authors but, on the whole, they all refer to the skills 
needed for employment and for life as a responsible citizen, 
which are important for all students regardless of the discipline 
they are studying (Fallows & Steven, 2000). As can be seen, 
competence refers to integrated behavior in which skills are an 
important component, as well as others such as knowledge, 
attitudes, rules and values. In this respect, Haste (2009, p. 207) 
states that “‘competence’ does not mean skills (although it may 
include skills).” 

Emphasis must therefore be placed on the integrative 
character of the concept of competence, in which different 
resources combine in order to respond to a given situation, as 
can be seen in Figure 1. 


Implication of competences 



Figure 1. The concept of competence at the University of Deusto (Pob- 
lete 2003, cited in Poblete & Garcia, 2007, p. 51) 

2.2 Competence assessment 

Assessment is normally one of the major sources of 
dissatisfaction among university students (Ferrell, 2012). 
Nonetheless, its importance in learning processes is vital. In the 
1970s a great deal of research already showed that what most 
influenced learning was not teaching but assessment (Boud & 
Falchikov, 2007). It is undeniable that assessment is currently of 
major importance to the university student and a key element 
that orients their learning, since students focus their attention 
and organize their way of studying by tailoring it to the type of 


assessment. (Villa & Poblete, 2011). Evaluation should therefore 
be carefully designed so that it becomes a precise tool that 
contributes to the attainment of deep learning. 

The transition from a traditional learning and teaching model 
to a competence-based model involves changes in the way of 
conceiving evaluation because traditional evaluation methods 
are no longer of use in assessing complex competences (Griffin, 
McGaw, & Care, 2012). 

In this new model, defining the competences we want the 
students to acquire is not enough - both the assessment and 
teaching methods must also be defined on the basis of the 
competences that wish to be attained. Succeeding in making 
assessment a strategy that encourages the development of 
students’ competences remains one of the greatest challenges 
facing university teaching. 

It is therefore necessary to rethink aspects such as what to 
evaluate, when, who, what for or how (Figure 2). 



Figure 2. Transition of the assessment model 

Aspects such as formative assessment, student participation in 
assessment processes, the use of methods that go beyond the 
traditional exam, etc., are by no means novel concepts. 
Nevertheless, their implementation in universities is proving 
complicated and their degree of integration is still rather limited. 

2.3 A model for generic competence assessment in 
higher education 

The University of Deusto has developed a generic competence 
assessment model that has been used as a reference point for 
designing competence assessment in the serious game now being 
developed. 

The generic competence assessment model includes three 
elements: 

Firstly , what one wishes to assess. The University of Deusto 
has defined 35 generic competences in this respect, grouped into 
three categories: instrumental competences, interpersonal 
competences and systemic competences (Universidad de Deusto, 
2001). This proposal of competences along with other initiatives 
by European universities was the seed that sprouted the study of 
generic competences undertaken by the Tuning Project 
(Gonzalez & Wagenaar, 2003), which was first led by the 
Universities of Deusto and Groningen and has spread further 
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afield to other European universities, where more than 200 
universities are involved, as well as other countries in Latin 
America and Asia. 

The University of Deusto's competence assessment model 
(Figure 3) provides the following information about each of the 
competences (Villa & Poblete, 2008, p. 51-52): 

—A definition that specifies and pinpoints the meaning that 
one wishes to be expressed under the term describing each 
competence. Such a definition solely intends to pinpoint the 
meaning given to it in this model, which may be different 
to how it is interpreted in other models or approaches. 

—The linking of competence with other closely-related 
competences , attitudes and values. Competence, as a human 
activity, is complex and is made up of skills, values and 
knowledge that cannot be separated from one another, nor 
can any claim full credit. Moreover, when one works in the 
learning of certain competences, it is wise to be aware of 
the fact that others are also developing. 

—The distinction of each competence in three levels of 
mastery. The first level reflects the competence's mastery 
within the person's normal context - a context where the 
person has to carry out their normal activities in familiar, 
everyday conditions. The second level indicates a mastery 
of the competence that displays good use of technique or 
competent, personal ways of acting. The third level refers 
to using the competence in different, complex situations. 
On the whole, it refers to situations of a professional nature, 
albeit not necessarily. 

—A number of indicators constituting clues or evidence of 
the competence's degree of development. 

—Each of the indicators is then fixed on a scale of five 
descriptors , which are easily translatable into traditional 
grades. 


Competences: 

• Generic 

• Specific 
Evaluated through 
indicators (learning 
outcomes) 



Figure 3. Generic Competence Assessment, University of Deusto 
(adapted from Villa & Poblete, 2008) 

An example of generic competence in accordance with this 
model is shown below (Table 1). 

Secondly , the competences assessment model requires explicit 
definitions to be given as to how generic competences will be 
assessed, i.e., what techniques and instruments will be used for 


the assessment and when they will be applied (at the beginning, 
during or at the end of a learning process) and who is going to 
take part in the assessment: only the teacher, the teacher with the 
student (shared assessment), the student by means of self- 
assessment and/or the students among themselves (peer 
assessment). 

Thirdly , a rating or weighting system for the final grade must 
be established, i.e., the weight each competence will have in the 
final rating. The model also allows different weights to be given 
to each competence's assessment indicators. 

The following figure (Figure 4) shows the University of 
Deusto's competence assessment system. 



Figure 4. Assessment elements, University of Deusto 

3 WHAT ARE SERIOUS GAMES AND WHAT DO 
THEY CONTRIBUTE TO LEARNING AND 
ASSESSING COMPETENCES? 

3.1 What are serious games? 

The game's overall function -and that of the video game in 
particular- as a tool for behavior and attitude learning, has been 
used in education for some years. When categorizing games, it is 
useful to consider the primary function of the game. Games are 
commonly played for fun and entertainment; however, some 
argue that they are played primarily to learn, even if it is with 
unconscious intent (Crawford, 1982). Serious games are a type 
of video game, characterized by being designed specifically for 
education, training and information purposes (Michael & Chen, 
2006). This does not mean they cannot be fun as well, but the 
game's amusement is designed for learning purposes so that 
student learning becomes enjoyable (Koster, 2004). 

The terms serious games and Game-based learning are 
sometimes used synonymously, although serious games have 
been developed for the broader purposes of training, and 
behavior and attitudes have changed in business, industry, 
marketing, healthcare, government and NGOs, as well as in 
education. On the other hand, commercial digital games are 
mainly developed for fun and entertainment, and so the 
distinctive features of serious games in relation to video games 
with a purely commercial aim could be summarized as follows 
(Marcano, 2008): 

—Serious games are oriented to education, skills training or 
the understanding of certain complex processes (social, 
political, economic, etc.) 

—They model real-life situations 
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Table 1. Example of Generic Competence: Problem Solving. Level 2 


COMPETENCE 


Problem solving 


SECOND LEVEL OF 
MASTERY: 


Using own experience and criteria to analyze the causes of a problem and construct a more efficient, effective 
solution 


INDICATORS 

1 

2 

3 

4 

5 



Has difficulty in seeing 

Identifies complex 

Makes a good 


Recognizing a complex 

Tc unahlp to 

and analyzing the 

problems, analyses 

analysis that 

Has comprehensive view, 

problem and being able to 

lij UllClUIv IU 

handle complex 
problems. 

complexity of a problem, 

them and 

includes priorities 

brilliantly recognizes the 

break it down into 

doesn’t manage to break it 

subdivides them 

and breaks down the 

parts of the problem and 

manageable parts 

down into manageable 

into manageable 

problem into 

how they are related. 



parts. 

parts. 

manageable parts. 


Contrasting sources of 
information and handling 
facts rigorously 

Doesn’t worry 
about the 
rigorousness of 
information. 

Needs help in checking 
reliability of sources of 
information and strength 
of facts. 

The facts he/she 
uses are rigorous 
and come from 
trustworthy 

sources. 

Handles hard facts 

and knows how to 
manage conflicting 
information from 

different sources. 

Provides noteworthy 
analysis of conflicting 
information drawn from 
different sources, and 
even contributes new 

sources. 

Having a method of 
analysis for identifying 
elusive underlying causes 
and evaluating their 
impact on problems 

Makes deficient 
analyses of 

causes. 

Identifies causes, but still 
doesn’t evaluate their 
impact on problems. 

Correctly follows a 
method to identify 
causes and 

evaluate their 
impact. 

Provides a good 
method of analysis 
for identifying 

causes. 

In addition to providing a 
good method of analysis 
for identifying causes, 
evaluates their impact 
with comprehensive 
view. 

Presenting solution options 
that are most often 
effective for resolving 
problems. 

Doesn’t present 
any solutions. 

Presents solutions, but 
they are not effective. 

Presents more than 
one effective 

alternative 

solution. 

Presents array of 
options with 
effective solutions. 

The options proposed 
feature diversity, rigor 
and internal logic. 





Applies the most 


Having good criteria for 
choosing between 
alternative solutions 

Lacks criteria. 


Correctly utilizes 

appropriate criteria 

Develops own criteria 

Doesn’t know 

Utilizes criteria 

the criteria offered 

for weighing 

that lead to selecting the 

how to justify 

inappropriately. 

for choosing a 

options and 

best of alternative 

his/her decision. 


solution. 

choosing the right 
solution. 

solutions. 


Devising a realistic plan of 
action and follow-up for 
applying the solution 

Doesn’t devise a 
realistic plan of 
action. 

His/her plan of action is 
realistic but lacks a 
follow-up plan. 

His/her plan of 
action is realistic 

and includes a 
follow-up plan. 

His/her plan of 
action and follow¬ 
up plan noteworthy 
for their quality. 

Noteworthy for the 
quality of his/her plan of 
action and follow-up 
plan. Foresees 
contingency plans. 


—The interests are declared in the content (political, 
economic, psychological, religious, etc.) 

Bearing in mind the definition given to the term serious game 
by Abt (1970), we have considered the following structural 
components for the formal modeling of our game design: 

—Objectives: must be clearly defined and known at all times 
by the player. In the context of an educational serious 
game, they will be explicit in the competences to be 
exercised. 

—Rules: this component will determine the order, rights and 
responsibilities of the players, as well as the objectives to 
be achieved by each player in order to reach the challenge 
they face. 

—Challenge: this determines when the game is completed. 
The player will face problems for which solutions will be 
sought. Once they all are solved, the challenge will be met. 
For the proposed serious game, partial and overall game 
closure criteria will be specified in learning outcomes. 

—Interaction: this is the component arising from the game 
mechanics and dynamics and will lead to all the 
experiences the player will enjoy. These will emerge 
continuously as the result of the immediate feedback the 


game provides and also show evidence of progress toward 
the final challenge (end game criteria). 

These serious games can be used to meet training objectives in 
knowledge areas as varied as defense, education, scientific 
exploration, health, public policy, strategic communications, 
emergency management, etc. 

3.2 What is serious games contribution to the learn¬ 
ing and evaluation of competences? 

Numerous studies exist in the area of video games and serious 
games that deal with their relationship with learning (Connolly, 
Boyle, MacArthur, Hainey and Boyle, 2012; Guillen-Nieto & 
Aleson-Carbonell, 2012). The latest advances in cognitive 
sciences broadly support the learning principles the games 
incorporate (Gee, 2003) and highlight the need for the 
experiential and active nature of learning for a protagonist who 
rises to the challenge of solving specific problems, and the 
context subsequently sends them real-time feedback on the 
process. From the standpoint of learning theories, Connolly et al. 
(2012) state that serious games are built under the principles of 
constructivism, situated learning, learning based on problem 
solving and cognitive learning. When analyzing game-based 
learning, Wu, Hsiao, Wu, Lin & Huang (2012) added that 


45 



Benzanilla M.J.; Arranz, S.; Rayon, A.; Rubio, /.; Menchaca, /.; Guenaga, M.; Aguilar. E. / New Approaches in Educational Research 
3(1)2014,42-51 


another of the most widely used principles in this type of game 
is the experiential principle. 

Modern theories of effective learning suggest that learning is 
most effective when it is active, experiential, situated, problem- 
based and provides immediate feedback (Boyle, Connolly & 
Hainey, 2011). In Games-Based Learning (GBL), students 
develop communication or behavior strategies to reach a goal by 
handling situations, taking into account elements of contrast and 
trying different approaches to see how the environment reacts to 
their actions. 

Suggestions that game players might actually be developing 
useful skills led to optimism that games might provide a useful 
and attractive new method of learning (de Freitas, 2006) and 
have a huge educational potential (Gee, 2003; Salen & 
Zimmerman, 2004; Squire & Jenkins, 2003). So much so that 
Prensky (2001) argues that video games could be the mechanism 
with the greatest learning potential ever known. Prensky 
postulates that while it is possible to adjust the content of video 
games so that they are more in line with the objectives of 
learning and socialization, the actual content of the game itself, 
with its rules and organization -beyond the content- enables a 
great deal of positive learning. 

Regarding the specific learning developed by video games, in 
addition to certain tasks and/or training, aspects can be seen in 
literature confirming that not only is content acquired and 
concepts understood with these kinds of games, but that they 
also promote the acquisition of certain skills and competences 
(Gee, 2009). The motivational components, the knowledge 
generated, accessibility, strategies, skills, active experimentation 
and learning based on the solving of specific problems that 
serious games can develop can make them an element with huge 
potential in the development of competences. 

Indeed, authors such as Gros (2008) believe that they 
encourage self-esteem and have a motivational factor, in 
addition to the chance to develop cognitive skills and strategies, 
such as problem-solving skills, decision-making, searching and 
organizing information, perceptual motor skills and abstract 
reasoning. Moreover, it can be added that they increase 
coordination capacity, spatial perception and broaden our field 
of vision (Prensky, 2001), which has an impact on reading and 
efficient handling in 3D environments. They also encourage 
socialization since they act as instruments for the transmission of 
behavioral patterns, values and favorable attitudes in certain 
contexts and they improve cognitive processes such as 
reasoning, memory and social interaction skills, especially with 
games of a strategic nature. 

Research shows that the clearest impact of video games' 
formative potential is shown by the acquisition of digital 
competences (Prensky, 2005). Other authors are more specific 
and state that the following are developed: communicative skill 
(Gros & Grup F9, 2004; Lacasa, Mendez and Martinez, 2008), 
resource management and reflective criticism (Sedeno, 2010), 
and negotiation and decision taking (Gros & Grup F9, 2004), the 
deployment of organizational, design and planning strategies is 
also pointed out (Gros, 2008), in addition to strategic reasoning 
(Sedeno, 2010). 

Prensky (2005) maintains that it is possible to learn many 
skills thanks to serious games: collaboration, decision decision¬ 
making under pressure, calculated risk taking, lateral thinking 
and ethical behavior. He also states that many current games 
“require the learning of difficult, complex skills to achieve goals 
that are quite a challenge, which will be of great use to players 
for life in the 21st century” (Prensky, 2005, p. 37). O’Neill, 


Wainess and Baker (2005) identify five types of demand of a 
cognitive nature in video games: content understanding, problem 
solving, self-regulation, communication and collaboration or 
teamwork. 

According to these criteria, the use of video games and serious 
games in particular in the development of competences has been 
made clear. In this sense, it is especially important to follow the 
principles put forward by Gee (2003), in particular; to offer 
information on demand and just at the right time, offer the 
possibility to customize the experience, make it easy for players 
to become content creators and not simply consumers, 
encourage progressive adaptation to the game's needs, 
implement a cycle of experience that repeats itself throughout 
the game, supervise, respect and maximize motivating elements 
and allow experimentation in the game under new identities or 
roles. 

4 A PROPOSAL FOR THE DESIGN OF A 
SERIOUS GAME TO DEVELOP TWO 
GENERIC COMPETENCES: PROBLEM 
SOLVING AND ENTREPRENEURSHIP 

4.1 Why we decided to use a serious game to devel¬ 
op competences? 

The proposed serious game has been designed with the aim of 
helping the student develop Entrepreneurship and Problem- 
Solving generic competences, since they are essential for the 
continuing education the student will need to pursue throughout 
their life. 

Games play an important role in students’ everyday lives, as 
can be seen by the interest they show in them in their leisure 
time. Thus, video games become an excellent tool as they 
provide different aspects of a game, such as entertainment, a 
psychological boost and competition and collaboration (Wolf, 
2003). Within this context, our proposal is to use a serious game 
to join real and virtual environments within the classroom, 
allowing students to actively reflect on the relationship between 
the game content and contexts of everyday activity (Gee, 2008). 
Figure 5 shows the main characters of the game within the Maze 
which gives entrance to University. 



Figure 5. Main characters 


Bearing this latest goal in mind, one of the most interesting 
concepts that help us understand the role of video games as 
cultural and educational instruments is situated cognition 
(Holland & Lave, 2001). From this point of view, cognition is 
narrowly linked to environment, action and experience. 
Furthermore, knowledge is acquired not only through the ability 
to perform an action, but also the opportunities offered by the 
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environment that make sense in specific contexts. These specific 
scenarios, combined with related rules defining the precise game 
dynamics, constitute the standpoint from which to analyze 
student activity. Moreover, games provide the ability to interpret 
and construct dynamic models of processes taking place in the 
real world. Thus, this game is presented as a representation of 
everyday experiences at the University of Deusto for our 
students, since the game adventure takes place at the university 
(albeit at night) and the graphic environment recreates 
particularly symbolic scenarios of it (see Figure 6). Gee (2003) 
assumes that a cultural universe involves images, history, 
principles or metaphors that capture what a particular group of 
students consider normal about a certain phenomenon, and so 
our University's context becomes suitable for the purpose of 
attaining our learning objectives. Video games provide the 
power to force their players to engage in different practices with 
a particular goal that could prove attractive to them, since they 
do so with a sense of purpose that they either generate or accept. 



Figure 6. Assembly Hall 


All games are composed of a set of rules restricting their 
action, forcing players to manage available resources. Thus, the 
rules of a game force the player to think and compare the game 
with what they already know about the fact that the game 
simulates. This is an important lesson for real life (Prensky, 
2002). These rules and pursued fun might seem to be two 
opposing terms, but the rules in video games are the most 
consistent source of amusement for the player (Juul, 2005) since 
they could be seen as the prohibition of something that students 
wish to do as players. Their combination in the form of specific 
configurations makes the strategies required to succeed in a 
game even more complex than the rules themselves. Thus, we 
can consider that the games force players to form their own 
simulated results from their own experience of the game (Del 
Castillo, Herrero, Monjelat, Garcia, & Checa, 2011) by 
developing the proposed competences in the serious game being 
described. 

The simulation that every serious game involves emphasizes 
learning processes that focus on how we learn rather than what 
we learn. Understanding learning through reflection and 
management, such as these serious games allow, better prepare 
the student to put them into action. The development of 
inductive and deductive thinking for the problem-solving 
competence can be worked on, generating new ideas and further 
testing. It is a way to prepare students to better handle change 
and unfamiliar situations. 

4.2 Project development process 

The serious game design for training in competences has been 
developed within the framework of the Bizkai Lab scheme, a 


scheme set up by the Provincial Council of Bizkaia that backs 
University of Deusto research projects. DeustoTech Learning, 
DeustoTech Computing, Institute of Leisure Studies and Innova 
research groups have joined forces to develop the project. 
Hence, an effective multidisciplinary group has been created to 
address the different dimensions of the project: the pedagogical 
and didactical framework, area of innovation and the technology 
and leisure perspective. The terms and conditions of the 
BizkaiLab scheme that enables the project to be developed 
stipulated a total of 3,300 hours for the entire team and a 12- 
month time frame. In this respect, the team had to adapt to the 
scheme's terms and conditions in order to ensure the project's 
viability and thus take decisions regarding design and 
development. 

This project aims to develop a serious game in order to 
exercise and assess generic competences. In this sense, in the 
initial phases of the project the team decided that the game 
would address the development of competences at levels 
corresponding to the final undergraduate and initial postgraduate 
academic years. 

These competences were not chosen arbitrarily; rather, the 
criteria summarized below were followed. Firstly, the team 
focused on the competences most regularly developed on the 
different degree courses in the final undergraduate and initial 
postgraduate academic years. Secondly, the “playability” of the 
competences was analyzed, i.e., the types of activities performed 
in the analyzed serious games were taken into account while the 
state of the art was being reviewed and, taking resources into 
consideration, some competences such as oral and written 
communication were ruled out. Finally, another selection 
criterion taken into consideration was the relationship of 
competences with the goal of employability, given that one of 
the aims within the terms and conditions of the BizkaiLab 
scheme was to develop projects designed to promote 
employability. In this respect, it was concluded that the two 
competences that best fulfilled such requirements were 
entrepreneurship and problem solving. 

Determining the target group also led to the analysis of the 
characteristics, taste and interests of students when it came to 
choosing their leisure experiences (see Figure 7). The task of 
defining the target group was essential because the scheme 
sought to respond to their needs and interests so that the intrinsic 
motivational factor in serious games could be tackled. 



Figure 7. Features of the game's target group 
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Table 2. Relationship between evaluated games, competences and indicators 

Scenario 

Mini game 

Competence 

Level of mastery 

Indicators 


1- Minotaur 

Game 

Problem¬ 

solving 

1. Identifying and analyzing a 
problem to generate alternative 
solutions by applying methods 
learned 

1.2. Reading and/or listening 
actively. Asking questions to 
define the problem at hand 

1. Maze 

Description: entrance to the University via 
the maze. Next to the door (closed), there 
is a visible alarm, with a light or similar to 
indicate their state 

2- Water and 
water pipes 
Game 

Problem¬ 

1. Identifying and analyzing a 
problem to generate alternative 
solutions by applying methods 
learned 

1.1. Identifying what is and is 
not a problem, taking the 
decision to address it 

solving 

2. Using own experience and 
criteria to analyze the causes of a 
problem and construct a more 
efficient, effective solution. 

2.1. Recognizing a complex 
problem and being able to 
break it down into 
manageable parts 


The fact that the project targeted university students in their 
final undergraduate and initial postgraduate academic years also 
enables the project's future experimentation and substantiation in 
a real scenario. It will therefore be crucial to establish, during 
the project's implementation phase, a complete assessment 
system providing information on the serious game's degree of 
usability, its formative capacity in competences, the students' 
degree of satisfaction and participant teachers' assessment. All of 
the assessments will be used to improve and adapt the serious 
game to the new needs detected. 

The project is divided into three phases and the team is 
currently in the project's third phase - the pilot development 
phase. A description of the objective and milestones reached in 
each of the phases is given in Figure 8. 


Multidisciplinary 
team was formed 


Analysis of the state 
of the art: definition 
<2 tj of Serious Games, 

® * features, pedagogical 
o £ functions, 

% possibilities, 

“> u technical 

-g requirements, etc. 


-f 


^ 00 Research on the 
hh possible 

technological tools 
available for the 
P-* technological 

development 


Study of the generic 
competences 
developed at the 
University of Deusto 

Selection of two 
S5 generic competences 
~ Study of the suitable 
o methodologies for 
Sr developing each of 
.. the selected 
^ competences 
s Definition of the 
jfj scenarios, activity 
types, interaction 
possibilities, 
assessment and 
feedback systems, 
etc. 

Specification of the 
pilot: the Serious 
Game's narration, 
scenarios, types of 
mini games, etc 


Implementation of 
the prototype 
Use of identifed 
technology' as most 
-g op suitable for the 
g 3 technical 
.. » development of the 
0 T 3 Serious Game 
k 73 Further technological 
j| •g development and 
^ p subsequent 

c_ experimentation and 
testing 

> 

a> 

T3 


Figure 8. Project phases 

4.3 Features of the serious game’s assessment sys¬ 

tem 

As mentioned above, a competence-based training model 
requires a rethinking of the assessment system in order to move 
from a traditional evaluative model to a model designed for the 
development of competences. In the design presented, we have 
attempted to include the competence assessment principles 
outlined above. Listed below are the principles we have 
highlighted as being important, together with the way in which 
they have been implemented in the game. 


Coherence between training and assessment: In a competence- 
based learning model, it is essential that coherence exists 
between the competences to be attained, the methods defined to 
achieve them and the assessment systems implemented. It is 
what Biggs & Tang (2011) calls aligned teaching. The game 
design set out from the competences and these have been the 
permanent reference point for the design of the formative 
activities and assessment system. In effect, as can be seen in 
Table 2, all the game's activities have a clearly established level 
of competence and working indicators. 

Assessing globality: A competence is made up of different 
elements (knowledge, skills, attitudes, motivation, values, etc.) 
and its assessment cannot be the result of adding up partial 
results for the different components in order to obtain a final 
rating (de la Orden, 2011; Villa & Poblete, 2011). It was decided 
that global activities would be incorporated into the game that 
allow several indicators from one competence to work, using 
cases, projects, etc., which gave a more holistic view of the level 
of achievement. 

Taking the context into account and transferring the 
knowledge to new situations: Competence assessment should 
be performed in the most familiar situation possible to the 
context in which the students develop (Fernandez, 2010; Villa & 
Poblete, 2011). Solving cases, problem-based learning, project- 
oriented learning and, of course, simulations and games can all 
be of use for such purpose as they help the student get their 
bearings in different scenarios and show their degree of 
competence. The students are given the chance to experiment 
with different topics, scenarios and cases in the game, and can 
sometimes even choose the topic to be addressed -we felt that 
this was important because the game is aimed at all the 
university students regardless of their different academic areas 
and the game had to adapt itself to different student profiles. 

Diversifying assessment methods and techniques: Given the 
importance of assessment in the way students organize their 
studies, the methods and instruments used for assessment will 
determine whether the students develop superficial or deep 
learning (Villa & Poblete, 2011). In the game, we have done our 
utmost to ensure that the assessment methods are as 
heterogeneous as possible, ranging from activities based on a 
system of questions and answers to the solving of cases or 
designing of projects. 
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Table 3. Example of a competence's score distribution 


Indicator 




Indicator/LO 

weight 

Mini game 

Who assesses 

Weight 



Evacuation of the assembly hall 

System 

10 

1.1. Taking initiative in daily situations 

20% 

Entrepreneurial project - 

System 

5 



brainstorming 

Peers 

5 

1.2 Weighing up risks and opportunities, taking decisions as 
accordingly 

15% 

Evacuation of the assembly hall 
Entrepreneurial project - the circle 

System 

Self-assessment 

5 

5 


closes in 

Peer assessment 

5 



Evacuation of the assembly hall 

System 

4 

1.3. Being able to anticipate the effects of actions undertaken 

10% 

Entrepreneurial project - the circle 

Self-assessment 

3 



closes in 

Peer assessment 

3 



Evacuation of the assembly hall 

System 

2 

1.4 Including social criteria in one’s own decision-making 

10% 

Entrepreneurial project - the circle 

Self-assessment 

4 



closes in 

Peer assessment 

4 



Shower of mortarboards 

System 

2 

1.5 Having enough self-esteem to take initiatives 

10% 

Entrepreneurial project - 

Lecturer 

Self-assessment 

2 

3 



communicating 

Peer assessment 

3 


Providing students with information on their progress: 

Feedback is an essential element of competence-based learning, 
whose purpose is to give the student continuous updates on their 
achievements and difficulties (de Miguel, 2006). This is perhaps 
one of the aspects that has received the most attention 
throughout the serious game's design process, seeking to provide 
the player with immediate, detailed, relevant and motivating 
feedback at all times. An example of feedback within the mini 
game “Evacuation of the Assembly Hall” in which students need 
to take decisions for solving everyday life problems (in this case 
related to team work) is the following: “Have you thought of 
how this decision will affect the clima of the team? and to its 
quality? ^and to the grades that each of you will get? It seems 
that this decision affects the team projects of several subjects. 
Are you sure you can take charge of his/her work in every case? 
Is this fair? What will happen in following projects?”. 

The intervention of different agents in the assessment 
process: An increasing number of authors agree that both self- 
assessment and peer-assessment, in addition to contributing to 
learning, help students learn evaluative skills, which they will 
need to develop both in their academic lives and careers (Biggs 
& Tang, 2011; Boud & Falchikov, 2006). These ways of 
actively participating in the evaluative process will enable them 
to be more self-critical of their degree of competence 
achievement, develop their autonomous learning competence 
and improve their determination thanks to their colleagues' 
feedback and also the effort involved in evaluating their peers 
and providing accurate feedback, as stated by Lu and Law 
(2012) “students benefit more as assessors than as assesses” (p. 
270). These processes also help to clarify the assessment criteria, 
add greater transparency to the process and open channels for 
dialogue and improvement. 

This point, which is perhaps one of the most novel and 
significant aspects in competence evaluation, is also 
undoubtedly the aspect that poses the most technical difficulties 
when being implemented in a serious game. In our case, 
although the game was initially designed with a markedly social 
character, owing to time and budget limitations, this part was 


unable to be included in the first phase of the game's 
development. Nonetheless, we believe that this peer interaction 
and assessment is extremely important, so face-to-face activities 
to be developed in the classroom have been designed which are 
integrated with the game and allow to take advantage of the 
richness of this social aspect. In any case, the design for the 
second version of the game, planned for 2013, includes small 
group work. This part requires group members (groups of 2 -3 
members) to participate synchronously, and asynchronously (but 
with time markers) between the different teams (4 or 5 per 
game). In order to do so, different rating and feedback systems 
have been designed to be used between teams and within a 
single team (for example, allocating fixed scores to different 
proposals put forward by other teams, voting within the same 
team, detailed explanations of the reason for deciding to join to 
an initiative put forward by others or not, etc.). 

All of this has led to a feedback and rating system that takes 
the competence indicators, games and assessment agents into 
account, as can be seen in Table 3, where the distribution of 
scores can be seen in level 1 entrepreneurship competence, 
which amounts to 65% of this competence's final rating. 

In addition to this global scoring system, a dual pedagogical- 
ludic script has been designed describing each mini game's aims, 
rules and interaction (Table 4). 

5 DISCUSSION 

Competences assessment is no easy task. Universities face the 
challenge of introducing the competence-based approach into 
their study programs, which is complex enough in itself because 
of the inclusive and systemic nature of the concept of 
competence. Moreover, the introduction of generic or transversal 
competences in the curriculums of different university degrees is 
new to teaching staff, who have to integrate these competences 
into their subjects' specific set of competences, apply 
methodologies, often different to those previously used, for 
competence learning and teaching, deduce assessment criteria 
and choose the techniques, times and assessment agents to do so. 
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Table 4. Sample of the dual pedagogical-ludic script 


_ Pedagogical script _ Ludic script 

Mini game _ Evacuation of the assembly hall 

Scenario _Assembly hall 


£ 


To develop entrepreneurship competence, in particular at its first 
level of mastery: Face everyday reality with initiative, weighing up 
the risks and opportunities and accepting the consequences. 


The university assembly hall must be rid of ghosts in order to 
retrieve a key needed to access the final scenario. 


Players must solve the different everyday life cases. In order to do 
so, they will need to take decisions by assessing certain aspects 
relating to the indictors to be assessed. 


To increase the degree of mastery in the following competence 
indicators (learning outcomes): 

1.1. Taking initiative in daily situations. 

1.2 Weighing up risks and opportunities, taking decisions accord¬ 
ingly. 

1.3. Being able to anticipate the effects of actions undertaken. 

1.4 Including social criteria in one’s own decision-making. _ 

Feedback plays a key role in the game as it should enable the player 
to analyze the decisions taken and assess the appropriateness of 
their answers. 

The decisions the person needs to make take all the selected indica¬ 
tors into account proportionally. Each indicator is also exercised in 
several of the cases in such a way that the player has the chance to 
exercise the competence and progressively show their progress. 


Certain problems preventing the ghosts from leaving need to be 
solved in order to empty the assembly hall. Each ghost puts a case 
to the player and asks them for advice on what to do to solve it. 
The player must take decisions and the system will give feedback 
on the appropriateness of their answer, which will build up differ- 
ent scores linked to each of the indicators. 

According to the decisions taken by players, they will either solve 
the problem posed by the ghost (and therefore rid the hall of it) or 
they will be allowed to analyze why their decisions were inappro¬ 
priate and will be given another chance to try out a new problem. 
Success is achieved in the game when the problems posed by all 
the ghosts in the hall are solved. 

The game provides detailed feedback on each of the answers 
chosen by the player. These answers are also rated according to 
each of the indicators. 

On finishing the mini game (or quitting it), the system provides 
the player with a summary table with the score the player has 
accumulated in each of the competence achievement indicators. 


In principle, serious games are an extremely valuable tool for 
exercising competences because they enable the creation of real- 
life or simulated learning scenarios in which the student must 
solve problems linked to their study or career area, they facilitate 
active learning, interaction and feedback, and they have a 
significant element of motivation. However, competence 
assessment often fails to be made clear in the game. Tasks, 
activities and mini game performance are assessed, but not 
exactly the performance level reached by a student in relation to 
a competence's indicators. This paper has endeavored to 
progress in this direction by presenting a clearly defined 
competence assessment model based on levels, indicators and 
descriptors that has been used as a benchmark for the assessment 
design of two generic competences: problem solving and 
entrepreneurship, in a serious game. 

The Competence Assessment Model described allows a better 
definition of the game scenarios and learning activities and 
provides a reference for a more operational evaluation of 
students performance through specifications of the competences 
to be evaluated, as well as a more comprehensive evaluation that 
the inherent complexity of a competence itself requires. 

This paper has analyzed what competence assessment consists 
of from a theoretical perspective, its contributions in relation to 
“traditional” assessment, the key elements coming into play 
(alignment, globality, context, feedback, agents, the grading 
system and so on) and the difficulties involved. Transferred to a 
serious game environment, it is true to say that the difficulties 
increase owing to the system's own limitations when assessing 
“performance” and integrated knowledge (knowledge, skills, 
attitudes and values). Even so, establishing indicators grouped 
into levels of difficulty for each competence (UD Competence 
Assessment Model) has been of great help in order to define 
learning and assessment activities and for coherence to exist 
between them (alignment), and they truly respond to what one 
wishes to achieve (integrity). In any case, it is thought that 


effective generic competence assessment should not be confined 
to the use of a serious game, but rather, that this will constitute 
one more element, a valuable resource, along with others, which 
will need to be used by the student within the framework of a 
more global-learning and teaching strategy. 

Important elements of competence assessment have been 
incorporated into the serious game presented in this paper. 
However, interesting elements are still to be introduced such as 
the game's collaborative dimension, self and hetero-assessment 
as empowering and motivating learning tools and the 
diversification of assessment strategies, which will be addressed 
in the second stage of the project. This will also enable us to 
address level 3 of the Problem Solving and levels 2 and 3 of 
Entrepreneurship competence, of which collaboration and 
personal leadership, to name just two, constitute an inherent part. 
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